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Design and Assessment of a Skills-Based Geoscience Curriculum 
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ABSTRACT 

There is increasing interest in science curricula that engage students in scientific practices to deepen their understanding of the 
core ideas of the field. There are, however, few examples of their designs or assessments available for use as guides for 
understanding the strengths and weaknesses of particular designs. We offer a summary of our experience in designing and 
assessing a skills-based geoscience curriculum contextualized within a recently proposed, generalized framework for 
curricular designs and assessment. Our novel approach to curriculum assessment, with its emphasis on ensuring that the skills 
highlighted in different courses align with the objectives of the curriculum, is meant to complement more-direct assessments 
of student mastery of various skills. Assessment results reveal good alignment of course and curricular objectives, in some 
cases, whereas other parts of the curriculum require refinement. The most challenging of which is the failure of the 
introductory-level courses to emphasize the process of science. That result is consistent witb recent work demonstrating that 
current approaches to teaching the nature of science are often ineffective. Thus, some of the most challenging problems with 
more traditional curricula, such as how to help students develop a more-advanced understanding of the nature of science, are 
not necessarily solved by focusing curricula on the integration of skills with content. © 2014 National Association of Geoscience 
Teachers. [DOI: 10.5408/13-100.1] 
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INTRODUCTION 

The geoscience education literature is replete with 
examples of the design and assessment of individual courses 
and even offers some examples of integrating specific skills 
(e.g., quantitative analyses, writing, or field-based instruc¬ 
tion) across multiple courses (e.g., Macdonald and Bailey, 
2000; Knapp et al., 2006; Leydens and Santi, 2006). Far less 
common, at least at the undergraduate level, are examples of 
curriculum design as used in this article, i.e., the design and 
sequencing of a series of undergraduate courses from 
introductory to advanced levels (Kelso and Brown, 2009). 
Almost nonexistent, in the geoscience literature, are 
quantitative assessments of undergraduate curricula. 

Beyond geosciences, considerable discussion of curric¬ 
ulum design has been exchanged; yet, at least at the 
postsecondary level, there has been limited quantitative 
curriculum assessment. Most existing quantitative assess¬ 
ments are focused on training in professional skills, 
particularly in the medical fields (e.g., Roxburgh et al., 
2008; Hall, 2014). Doll (1993) argues that most of those 
evaluations are "modernistic" because they seek to deter¬ 
mine the extent to which a curriculum achieves well-defined 
educational goals, with learning a specifically intended, 
directed, and controlled outcome—one that can be mea¬ 
sured. That approach to curriculum design and assessment 
has been criticized for reducing the complexity of the 
curriculum to easily measurable objectives that primarily 
address lower-order behaviors, rather than higher-order 
thinking. As an alternative, Doll (1993) argues that 
postmodern curricular development should follow Dewey 
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(1938) and emphasize the exploration of the unknown, 
allowing educational ends to arise from within the process of 
experiential activity, with learning being a byproduct of that 
activity (Koo Hok-chun, 2002). This approach to curricular 
design is consistent with the Hargreaves (1994) argument 
that the skills required to succeed in the postmodern world 
include inquiry, information gathering, analysis, and other 
aspects of learning how to learn in an engaged and critical 
way. The Doll (1993) postmodern approach to curricular 
development has been criticized for being overly vague, 
difficult to assess, and, to a certain extent, unrealistic, 
idealistic, and impracticable (Koo Hok-chun, 2002). 

The Next Generation Science Standards (NGSS) might 
be seen as an attempt to reconcile modern and postmodern 
approaches to curriculum development by explicitly linking 
core disciplinary ideas and crosscutting concepts with 
scientific practices, where the term practice is used rather 
than skill to stress that engaging in scientific inquiry requires 
the use of both knowledge and skill simultaneously (NAS, 
2012). The NGSS is a flexible model for curricular design, in 
which the focus is not on particular courses and their content 
(as typical of a purely modernistic model) but, rather, is on 
the integration of knowledge and skills across the curricu¬ 
lum, with content learning being a consequence of directed 
experience in scientific practices (as emphasized in the 
postmodern model). In the NGSS model, the curriculum 
becomes more than just a course-by-course description of 
core content to be learned and, instead, complements 
required core content with a description of the skills or 
practices expected of all students by the completion of their 
degrees. Although grounded in foundational ideas, the 
emphasis of the NGSS on scientific practices allows for an 
exploration of the unknown, with the specific educational 
outcomes arising from activities that allow the student to 
practice the skills required for students to quickly adapt to 
fields, both within geosciences and beyond, where new ideas 
and techniques rapidly emerge and where tenure at any one 
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position is often limited (Manduca et al., 2008). Although the 
generality of a skills-based curriculum makes it easier to 
adapt to a variety of student needs, it also presents a 
secondary challenge beyond its design; namely, how is it 
assessed? The question of assessment is particularly impor¬ 
tant given the increasing demands of governmental and 
accrediting agencies for greater academic accountability 
(Middaugh, 2007). 

The idea of skills-based geoscience curricula at the college 
level is not necessarily novel, but there are few examples in 
the literature of their design or their assessments for use as 
guides to understanding the strengths and weaknesses of 
particular programs. In that spirit, I offer a brief summary of 
our experience in designing and assessing a skills-based 
geoscience curriculum contextualized within the framework 
of a recently proposed model for curriculum design and 
assessment (Hall, 2014). Although I briefly describe the 
development of our curriculum, the emphasis here is on the 
novel approach to assessment. That assessment is an ongoing 
process, and thus, I present here a status update rather than a 
complete case study. Our assessment efforts, however, have 
already provided important feedback regarding how our 
curriculum can be improved, which aligns with recent work 
suggesting that current approaches to teaching the nature of 
science may not be effective (Abd-El-Khalick, 2006; Nadelson 
and Viskupic, 2010). 

CURRICULUM DESIGN 

Motivation 

By many metrics, Dartmouth already had a strong 
undergraduate geoscience curriculum before our curricular 
redesign. During the course of this study, we had a faculty of 
10, averaged 10-15 majors per year split between two 
majors, and supported about 20 graduate students. Almost 
all classes were taught by regular faculty, with teaching 
assistants taking on only limited teaching roles in introduc¬ 
tory laboratory sections. We offered (and continue to offer) 
almost all core geoscience courses most frequently provided 
by geology departments in the U.S. (Drummond and 
Markin, 2008). In addition, Dartmouth is among the top 25 
U.S. institutions in producing undergraduates that go on to 
earn doctorates in geosciences, both in absolute numbers of 
undergraduate majors going on to earn doctorate degrees 
and in "institutional-yield ratio" (Fiegener and Proudfoot, 
2013), i.e., the number of doctorates earned normalized by 
the total undergraduate enrollment (see Appendix A). 

By the early 2000s, our traditional, prescribed-course- 
based (i.e., "modernistic") curriculum was becoming in¬ 
creasingly unwieldy as the number of sub-disciplines and 
skills needed by our graduates increased and evolved 
(Heath, 2003) and the careers pursued by our graduates 
diversified. The distribution of careers among our alumni is 
generally similar to national averages (AGI, 2001) but with a 
greater fraction of our alumni pursuing nongeoscience 
careers when compared with the national average (about 
30% versus about 10% nationally). 

Because the number of courses required in a major is 
more or less fixed, increasing the number of required core 
courses as the discipline expands leads to a reduction in the 
number of elective courses a student can take, reducing the 
flexibility that allowed students to adapt their course 
selections to their career interests. Also, focusing on 


Design and Assessment of a Skills-Based Curriculum 669 

individual core courses does not, by itself, shed light on 
how the different courses interact pedagogically. This latter 
point is important because of increasing demands from 
institutional and accrediting agencies for clear statements of 
anticipated student learning outcomes not only at the course 
level but also at the department level, where learning 
outcomes are expected to be aligned with each other. 

Framework for Curricular Design 

Recently, Hall (2014) proposed a framework for curric¬ 
ular design and assessment based on the earlier work of 
Glatthorn (1987) and Print (1993). That framework envi¬ 
sioned the process of curriculum design and assessment to 
involve four key steps: the first two focused on design the 
latter two on assessment. In Step 1— benchmarking —key 
curricular objectives identified by professional bodies and 
peer institutions are reviewed. In Step 2— evidencing —a 
correspondence is established between the benchmark 
standards and the proposed curricular elements. In the first 
of the two assessment steps, knowing —the congruence 
between the planned and delivered curriculum—is assessed. 
In the Step 4— applying —the extent to which students are 
achieving the curricular objectives is assessed. Those steps 
generally align with the four foundational questions for 
curriculum design originally proposed by Tyler (1949): 

1. What educational purposes should the curriculum 
seek to achieve? 

2. What learning experiences are useful in attaining 
these objectives? 

3. How should these learning experiences be orga¬ 
nized? 

4. How can the learning experiences be evaluated? 

Although we were unaware of that framework for 
curricular design when we revised our curriculum, a post hoc 
review of our process reveals that it generally followed the 
framework. 

As envisioned in the framework, we began our 
curriculum design with a process of benchmarking against 
peer institutions and professional bodies. Specifically, to 
define the sets of key concepts and skills that would define 
our curriculum, we used a "backward design" (Wiggins and 
McTighe, 2005) approach in which we started by outlining 
what a graduate of our program should know and be able to 
do. For guidance, we grouped common concepts and skills 
that appeared repeatedly in geoscience program-assessment 
instruments used at peer institutions (available at the 
Science Education Resource Center, http://www.serc.edu). 
Ultimately, we identified five categories of key concepts and 
skills and assigned each of our courses into one of those 
categories (Table I). We then defined the curriculum so that 
all majors had to take at least one course in each course 
category, thus ensuring that each student took at least one 
course placing particular emphasis on each set of concepts 
and skills. This approach to curriculum design might be 
viewed as a hybrid between modern and postmodern 
approaches. Consistent with the dictates of postmodern 
design, the curriculum does not define specific content to be 
learned. Instead, the specific content learned is a byproduct 
of students practicing specific skills and concepts in the 
courses they elect. However, consistent with modern 
curricular design, our skills-based curriculum has well- 
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TABLE I: Student learning goals and laboratories for the Cyrosphere and Earth Systems Science Earth Laboratories modules. 


Key Concepts and Skills 

Example Courses 

Introductory Courses ( one course required) 

Concepts of geologic time and geologic scales 

How the Earth Works 

Earth's origins 

Environmental Change 

The use of maps and spatial data 

Evolution of Life and Earth 

The process of science and the evolution of scientific concepts 


Earth as a dynamic system including physical, chemical, and biological interactions 


The impact of geology and natural resources on the evolution of life and vice versa 


The concept of uncertainty as it applies to our understanding and analysis of Earth 
processes, resources, and hazards. 


Collection and Analysis of Earth Science Data (one course required) 

Collection and interpretation of earth science data 

Earth's Climate: Past, Present, Future 

Collection of field observations 

Meteorology 

Quantifying uncertainty 

Environmental Geology 

Core Methods and Concepts ( 4-6 courses required) 

Origin of earth materials 

Mineralogy 

Mineral transformations 

Structural Geology 

Structure and mechanics of earth materials 

Sedimentary Systems 

Co-evolution of Earth and life 


Geotectonics 


Core field methods 


Working in groups 


Quantitative Analysis of Earth Systems ( one course required) 

Simplification/modeling of complex systems for quantitative analysis 

Geophysics 

Quantitative analysis of earth systems 

Hydrogeology 

Limits of data/knowledge 

Geochemistry 

Advanced Topics ( one course required) 

In depth exploration of a specialized topic at an advanced level 

Environmental Isotope Geochemistry 

Reading the scientific literature 

Advanced Quaternary Paleoclimatology 

Abstract writing 

Geobiology 


defined educational goals (i.e v the mastery of specific skills) 
that are, in principal, measureable. 

Although developed independently, our key concepts 
and skills categories are consistent with the U.S. National 
Research Council's Framework for K-12 Science Education 
(FSE), which is the basis for the NGSS. As with our 
approach, the FSE does not seek to define course structure 
but, rather, to define a vision for science education. In that 
vision, students, during multiple years of school, engage in 
scientific practices that deepen their understanding of the 
core ideas in a field (Quinn et al., 2012). Consistent with the 
postmodern approach, in emphasizing practices over con¬ 
tent, the FSE seeks to help students understand that theory 
development, reasoning, and testing are components of a 
larger ensemble of activities defining the nature of science, 
which includes teamwork, specialized ways of communicat¬ 
ing, the development of models to represent systems or 
phenomena, the making of predictive inferences, and the 
testing of hypotheses by experiment or observation (Quinn 
et al., 2012). To achieve that vision of science education, the 
FSE identifies eight core practices intended to develop 


students' understanding of the nature of science and 
engineering. Those core practices include "planning and 
carrying out investigations" and "analyzing and interpreting 
data," both of which are consistent with the skills that define 
our Collection and Analysis of Earth Science Data course 
category (Table I). Similarly, the FSE core practices of 
"developing and using models" and "using mathematics and 
computational thinking" align with the skills that define our 
Quantitative Analysis of Earth Systems course category. 
Finally, the FSE core practices of "asking questions and 
defining problems" as well as "engaging in argument from 
evidence" are central to the "process of science and the 
evolution of scientific concepts" element of our introductory 
level course category. Demonstrating the congruence 
between our curricular design and the FSE represents the 
second step in the curricular design framework—evidencing. 

ASSESSMENT DESIGN 

Within the framework for curricular design and assess¬ 
ment described above, opportunities for assessment are 
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TABLE II: Explanation and scoring system for the conceptual understanding rubric. 


Questions 

Course Category Most Emphasizing Skill 1 

Question 1 


This course improved my quantitative and analytical skills 

Quantitative Analysis of Earth Systems (p = 0.020) 

Question 2 


This course improved my ability to communicate scientific concepts, 
data, and results in written and oral form 

Advanced Topics (p = 0.099) 

Question 3 


This course improved my familiarity with sources and the use of 
scientific literature 

None (see text) 

Question 4 


This course improved my familiarity with the process of science 

Collection and Analysis of Earth Science Data (p = 0.089) 

Question 5 


This course improved my ability to obtain, analyze, and interpret 
scientific data 

Collection and Analysis of Earth Science Data (p = 0.027) 


1 Misconceptions may or may not be present in levels 0-3. 


contained within the final two steps, knowing —determining 
the congruence between the planned and delivered curric¬ 
ulum—and applying —determining whether students are 
achieving the curricular objectives. Determining whether 
students are achieving curricular objectives is sometimes 
done using postcourse surveys or capstone exams. Such an 
approach, however, can be inefficient and redundant. If a 
course emphasizes a particular skill, and if the student's 
summative evaluation (e.g., final exam or project) in that 
course reflects their mastery of the emphasized skill, then 
the summative evaluation from that course presents a richer 
assessment of a student's mastery of that skill than can be 
determined using a postcourse survey. Thus, we approached 
the assessment of our curriculum by focusing on the first of 
the two assessment steps, the congruence between the 
planned and delivered curriculum. In our case, our 
hypothesis was that courses within a particular category 
emphasized the skills that define that category. 

Rather than relying on course syllabi to determine what 
skills a particular course emphasized, we instead opted to 
directly ask the students at the completion of each course 
which skills they thought the course emphasized. We 
recognized the potential for a student to misinterpret the 
purpose of a course. However, because clarity of goals is an 
essential element of effective teaching, such a misunder¬ 
standing, in most cases, likely indicates a failure of the 
course to serve its curricular purpose. Misinterpreting the 
goals of a course also likely represents a failure of the 
summative evaluation of the course to reflect a student's 
mastery of the targeted skills. We also recognized that the 
perception of emphasis on a skill does not imply mastery of 
that skill, a situation that should be reflected in the student's 
summative evaluation, but, again, our objective in this 
assessment was to investigate the alignment of course and 
curricular objectives. 

To determine which skills students thought a particular 
course emphasized, we added five questions to the online 
course evaluation survey that students are asked to complete 
at the end of each course (Table II). The five questions asked 
the student to what extent they agreed that the course 
improved their familiarity or ability in that skill. Students 
ranked each skill on a five-point scale, ranging from strongly 


agreeing that the course improved their familiarity or ability 
in that skill (ranking = 1) to strongly disagreeing that the 
course improved their familiarity or ability in that skill 
(ranking = 5). Students could also select not applicable. We 
cautioned the student that courses were designed to focus 
on particular skills, so not all skills listed may have been 
addressed. Some caution was also warranted in interpreting 
the results from that survey because no attempt was made to 
formally validate the skill-ranking questions. 

In the analysis of our data, I also used one item from the 
standard course evaluation. That question asked the student 
to rate the overall quality of the course on a five-point scale, 
ranging from excellent (rating =1) to poor (rating = 5). 
Separate analyses of the student responses to other 
questions on the course evaluation that addressed specific 
aspects of course quality, such as the extent to which course 
objectives were clear, assessment of how much was learned 
in the course, and course organization, were found to be 
highly correlated with assessment of the overall course 
quality. Thus, only the overall assessment of course quality 
was used here. 

Between fall 2009 and spring 2013, end-of-class online 
evaluations containing the five skill-ranking questions were 
given in 86 different classes. Except for the introductory 
course category, the distribution of courses offered between 
the different course categories was similar, with 10-20 
courses in each category. There were 29 courses offered in 
the introductory course category. 

Students completed the online course evaluations 
outside of class anytime within a window that extended 
from the last week of the term to the fourth day of the next 
term. Students were not required to complete the evalua¬ 
tions; however, those that did complete the evaluation could 
see their course grades earlier than those that did not 
complete the evaluations. 

The online course evaluations had four or five sections, 
each with five to 10 questions. The first section asked about 
student background, the second section about course design 
and effectiveness, the third section about faculty effective¬ 
ness, the fourth section about laboratory sections (if 
applicable), and the fifth section contained the five skill¬ 
ranking questions. 
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TABLE III. Summary of skill rankings for each course category. 


Course Category 1 \Skill 

Quantitative 2 

Communicate 

Literature 

Nature of Science 

Data Analysis 

Introductory (29/1,981) 

2.33 (0.38) 

2.20 (0.31) 

2.38 (0.35) 

2.13 (0.31) 

2.20 (0.39) 

Data analysis (16/182) 

1.83 (0.35) 

1.60 (0.38) 

2.08 (0.67) 

1.58 (0.35) 

1.44 (0.33) 

Core methods (18/217) 

2.12 (0.80) 

1.92 (0.31) 

2.19 (0.41) 

2.07 (0.80) 

1.84 (0.28) 

Quantitative (12/136) 

1.66 (0.25) 

1.74 (0.30) 

1.96 (0.45) 

1.79 (0.23) 

1.83 (0.57) 

Advanced (11/145) 

1.58 (0.41) 

1.65 (0.57) 

1.66 (0.39) 

1.56 (0.39) 

1.54 (0.55) 


Values in parentheses indicate (number of courses/total number of students). 
2 Mean (SD), lower mean values indicate greater emphasis. 


RESULTS 

Responses 

The 86 different classes given the skill-ranking questions 
had a total enrollment of 2,661 students. The number of 
courses and total number of students in each course category 
are summarized in Table III. Survey completion rates 
averaged 88% and ranged from a low of 77% in the 
advanced topics courses to a high of 90% in the introductory 
courses. 

Average student rankings ranged from 1.44 to 2.38 and 
are summarized in Table III. Lower-ranking values indicate 
the student more-strongly agreed the course improved their 
familiarity or ability with that skill. The average ranking for 
each skill across all courses were similar, except for the skill 
of using the scientific literature (single-factor ANOVA, p = 
0.002). Familiarity with sources and the use of scientific 
literature was, on average, slightly less emphasized in all 
courses relative to the other skills. 



FIGURE 1: Average perceived emphasis on skill ranking 
for courses offered in the years 2009-2013 as a function 
of reported course quality. Each point indicates the 
average values for an individual course. Symbols 
indicate rankings for different categories of courses. 
Solid line is the best linear fit for the entire data set (r 2 = 
0.34). 


Course Quality Effect 

Across all courses, the rankings of the skills were 
correlated (r 2 = 0.34) with overall course quality (Fig. 1). 
Higher-quality courses were generally ranked higher across 
all skills than lower-quality courses. To remove the effect of 
course quality on the relative rankings, only the residuals of 
the rankings were analyzed when comparing different 
categories of courses. To do that, for each skill, a best linear 
fit of course quality versus rankings was determined, and the 
residuals were determined as the difference between each 
ranking and the best linear fit. To determine whether a 
particular category of courses placed significantly more or 
less emphasis on a particular skill, single-factor ANOVAs 
were performed to compare the residuals among the 
different categories of courses. None of the ANOVAs were 
significant when all the categories of courses were consid¬ 
ered. Closer inspection of the mean and variance of the 
residuals for each course category revealed that, although 
the mean residual of the different skills rankings for courses 
in the Core Concepts and Methods category was typically 
close to zero (indicating that those types of courses, on 
average, tended not to emphasize a particular skill more or 
less than other types of courses), the variance of the residuals 
for Core Concepts and Methods courses was, in some cases, 
more than a factor of three larger than the corresponding 
variances of the residuals for other course categories. Thus, 
the particular skill emphasized by Core Concepts and 
Methods courses varied more from course to course within 
that category in comparison to other course categories. 

Removing the Core Concepts and Methods courses from 
the analysis, the single-factor ANOVAs for four of the five 
skills were significant (p < 0.05), indicating that each of 
those four types of skills was significantly more or less 
emphasized in at least one category of courses (Table II). The 
exception was that no category of courses placed particularly 
more or less emphasis on the use of scientific literature (p = 
0.26). Taken together with the above result indicating that 
across all courses the use of scientific literature was slightly 
less emphasized relative to the other skills, these results 
indicate a failure of the courses in our curriculum to 
appropriately address that skill. 

Contrast Analyses 

For the four skills with statistically significant ANOVAs, 
follow-up contrast analyses were done to compare, for each 
skill, the residuals of the rankings within each course 
category. Specifically, for each skill type, we used a single¬ 
factor ANOVA to compare either the highest or lowest 
average residual across the different course categories to the 
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Emphasis on Process of Science 

0.6 h 


0.6 


Emphasis on Data Analysis 


0.4 


0.2 

0 


** 



- 0.2 


- 0.4 




- 0.6 


Introductory Data Analysis Quantitative Advanced 

T opics 



Introductory Data Analysis Quantitative Advanced 

T opics 


Emphasis on Communication 

0.6 |- 


Emphasis on Quantitative Analysis 

0.6 |- 



Introductory Data Analysis Quantitative Advanced 

T opics 



Introductory Data Analysis Quantitative Advanced 

T opics 


FIGURE 2: Residual skill rankings for the different course types after the effect of course quality was removed. Lower 
rankings indicate students more-strongly agreed that course type improved their familiarity or ability with the given 
skill. ANOVAs on the residuals for each skill type shown are significant (p < 0.05). Follow-up contrast analyses 
(single-factor ANOVAs) compared either the highest or lowest average residual across the different course categories 
to the combined residuals of the other course categories for that skill. Course types with average residuals 
significantly different (p < 0.05) than the other course types are indicated by two asterisks (**). Marginally significant 
differences (p < 0.1) are indicated with a single asterisk (*). 


combined residuals of the other course categories for that 
skill (Fig. 2). 

Consistent with the design of the curriculum, students 
report that the Collection and Analysis of Earth Science Data 
courses placed significantly greater (p = 0.027) emphasis on 
obtaining, analyzing, and interpreting scientific data. In 
contrast, our introductory level courses placed significantly 
less (p = 0.011) emphasis on the analysis of scientific data. 
Also consistent with the design of the curriculum, students 
reported that Quantitative Analysis of Earth Systems courses 
placed significantly greater (p = 0.02) emphasis on 
quantitative and analytical skills. Here, too, introductory 
level courses were reported to place significantly less (p < 
0.001) emphasis on quantitative and analytical skills. The 
final two significant contrast effects revealed that, relative to 


our other types of courses, students report that our 
introductory level courses placed significantly less emphasis 
on the communication of scientific concepts, data, and 
results in written and oral form (p = 0.006) and on the 
process of science (p = 0.007). The Advanced Topics course 
was marginally significant for placing more emphasis on 
communication of scientific concepts, data, and results (p = 
0.099), and Collection and Analysis of Earth Science Data 
courses was marginally significant in placing more emphasis 
on the process of science (p = 0.089). 

DISCUSSION 

The assessment results indicated that although some 
aspects of our curriculum were working as intended, other 
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parts required refinement. Among the successes was the 
alignment of the skills emphasized in the Collection and 
Analysis of Earth Science Data and Quantitative Analysis of 
Earth Systems categories with the skills that defined those 
categories. That the skills those courses emphasized were not 
particularly emphasized in our introductory level courses can 
also be viewed as a success of the curriculum. That is not to 
say that data analysis and quantitative skills should not be 
addressed in introductory level courses, but rather that, 
according to our curriculum design, our introductory level 
courses should place more emphasis on understanding the 
process of science and the evolution of scientific concepts and, 
relatively, less emphasis on data or quantitative analyses. 

That the skills emphasized in Core Methods and 
Concepts courses varied from course to course was also 
consistent with the design of the curriculum. That is, those 
courses were not designed to place particular emphasis on 
any one of the skills included in our assessment. For 
technical reasons, we were limited to adding only five 
questions to the online course evaluation. If we could have 
added a sixth question, we would have asked the students to 
rank the relative emphasis placed on working in groups, a 
key skill defining the Core Methods and Concepts category. 

Among the areas of the curriculum that, according to the 
assessment results, needed correction was the failure of the 
Advanced Topics courses, or any other category of course, 
for that matter, to emphasize the use of scientific literature. It 
is difficult to argue that our students are mastering that 
critical skill if they report that, on average, none of their 
courses placed particular emphasis on in its importance. That 
deficiency could be addressed by reminding our faculty to 
include greater attention to the scientific literature in our 
Advanced Topics courses. 

Somewhat more challenging to address is the failure of 
Collection and Analysis of Earth Science Data courses to 
emphasize the communication of scientific concepts, data, 
and results in written and oral form despite scientific 
discourse being a key skill defining that course category. 
The idea that scientific discourse, rather than being 
emphasized only in one particular category of courses, is 
instead emphasized in all categories of courses is rebutted by 
the fact that the average ranking for this skill is no different 
than the average rankings of the other skills (except, as 
noted earlier, scientific literature). Interestingly, the assess¬ 
ment reveals that Advanced Topics courses were marginally 
significant for placing more emphasis on scientific commu¬ 
nication. That may reflect partially successful attempts in 
those courses to follow the dictates of the curriculum and 
emphasize familiarity with sources and the use of scientific 
literature. This focus was, at best, only partially successful 
because students did not rank Advanced Topics courses as 
particularly affecting their familiarity and use of the scientific 
literature (Table II). However, to the extent that the primary 
scientific literature is discussed in those courses, students 
may rank those courses as placing somewhat more emphasis 
on science communication. To address the lack of emphasis 
on scientific communication in the curriculum, we could 
decide either to place greater emphasis on scientific 
communication in our Collection and Analysis of Earth 
Science Data courses or, alternatively, to move that skill to 
the Advanced Topics courses and give it greater attention in 
those courses. That Advanced Topics courses already gave 
some emphasis to scientific communication suggests that 


moving the skill to the Advanced Topics category might be 
appropriate. 

Perhaps the biggest challenge indicated by the assess¬ 
ment results was the failure of the introductory courses to 
emphasize, at least as perceived by the students, the process 
of science. We recognize that students can sometimes be 
unaware of the positive effect an exercise or course has on 
their mastery of a particular skill (Renshaw and Taylor, 
2000), and thus, students in our introductory courses may 
not have recognized the effect those courses had on their 
understanding of the process of science. However, that 
assessment result is consistent with recent work demon¬ 
strating that current approaches to teaching the nature of 
science are often ineffective, resulting in students retaining 
naive conceptions on the nature of science, even as college 
graduates (Abd-El-Khalick 2000; Moss et al., 2001; Abd-El- 
Khalick 2006; Nadelson and Viskupic 2010). A potentially 
important distinction in terminology is worth noting. In 
designing and assessing our curriculum, we were consistent 
and always described the skill as "familiarity with the 
process of science." In the science-education literature, the 
idea we imagined in creating the curriculum is referred to as 
the "nature of science." Schwartz et al. (2012) argue that the 
two terms are not strictly synonymous. That is, in speaking 
of the process of science as a skill, they argue, we are really 
referring to what they term the "nature of scientific inquiry," 
rather than the broader suite of elements that make up the 
nature of science, which includes, for example, its tentative, 
creative, and subjective aspects. The distinction between 
terminologies is important for understanding the strengths 
and weaknesses of different approaches to teaching the 
nature of science (Khishfe and Abd-El-Khalick, 2002; 
McDonald, 2010), but the subtlety between the phrases 
process of science and nature of science is likely not perceived 
by the students completing the skill-ranking survey. 

In many cases, students failed to develop an under¬ 
standing of the process of science in introductory courses 
partly because those courses often emphasize science 
content knowledge more than they do the nature of science 
(Abd-El-Khalick, 2000; Bickmore et al., 2009). However, 
emphasizing content over process may not result in effective 
learning because a fundamental understanding of the nature 
of science is critical to learning science (Alters, 1997; NRC, 
1998; Nadelson and Viskupic, 2010). 

There is some suggestion that students can gain an 
understanding of the process of science through implicit 
exposure to the construct when engaging in scientific 
activities (Ryder, et al., 1999). This may explain students' 
perceptions of the marginally significant more emphasis 
placed on the process of science in Collection and Analysis 
of Earth Science Data courses. By engaging in the analysis of 
data, students may be gaining an implicit understanding of 
the process of science. If the understanding of the process of 
science is linked to engaging in scientific practices, it may be 
unrealistic to set the understanding of the process of science 
as a goal for introductory courses. Instead, students develop 
an understanding of the process of science as they proceed 
through a curriculum that emphasizes the different elements 
of scientific practice. Some support for this idea comes from 
our recent survey of alumni, in which almost three-quarters 
of the nearly 300 respondents reported that Dartmouth 
provided effective training in understanding the process of 
science. 
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However, other studies have shown that the assumption 
that engaging in scientific practices will develop one's 
understanding of the nature of science is flawed (Khishfe 
and Abd-El-Khalick, 2002; Schwartz and Lederman 2008). 
For example, Nadelson and Viskupic (2010) found a poor 
correlation between the number of courses taken by 
undergraduate geoscience students and their understanding 
of the nature of science. Instead, research during the past 2 
decades has found that the nature of science is best taught 
using an explicit and reflective approach (Abd-El-Khalick et 
al., 1998; Abd-El-Khalick and Akerson, 2004; McDonald, 
2010 ). 

An alternative explanation for why students indicated a 
marginally significant emphasis on the process of science in 
Collection and Analysis of Earth Science Data courses is that 
the nature of those courses may only be reinforcing students' 
naive "logical positivist" conceptions on the nature of 
science (Bickmore et al., 2009). That is, in analyzing data, 
students may be reinforcing their perception that the process 
of science is solely focused on the use of data to verify 
hypotheses, rather than the more-modern view that data are 
meant to falsify hypothesis. Further, students' limited focus 
on the data analysis component of the process of science 
may indicate that they fail understand that abstract 
reasoning, tentative conclusions, and creativity are all 
equally important aspects of the nature of science (Bickmore 
et al., 2009). 

CONCLUSIONS 

The vitality of the geosciences is partly attributable to 
their dynamic, integrative, and expansive nature (Gabler and 
Frank, 2005). Those traits result in an evolving science, and, 
as the science evolves, so too must the curriculum. With 
most geoscience undergraduates not planning to pursue 
careers in the traditional mining and energy industries 
(Wilson, 2012), the traditional curriculum must evolve to 
accommodate those students interested in nontraditional 
careers. Some guidance in how geoscience curricula should 
evolve is provided by the NGSS and its focus on the 
integration of knowledge and skills over course-by-course 
descriptions of core content to be learned. Currently, 
however, there are few examples of the implementation or 
assessment of a skills-based curriculum at the undergraduate 
level. 

The novel approach to curriculum assessment presented 
here, with its emphasis on ensuring that the skills different 
courses emphasized align with the objectives of the 
curriculum, is meant to complement more-direct assess¬ 
ments of student mastery of different skills. Determining 
whether the designs of different courses align with the 
dictates of the overall curriculum is an important first check 
of curriculum integrity. 

The assessment results indicated that some aspects of 
our curriculum were working as intended. Students reported 
that the skills emphasized in the Collection and Analysis of 
Earth Science Data and Quantitative Analysis of Earth 
Systems course categories aligned well with the skills that 
defined those categories. 

The assessment results also indicated that some the 
areas of the curriculum needed to be refined. The most 
challenging of which was the failure of the introductory 
courses to emphasize, at least as perceived by the students, 
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the process of science. That result is consistent with recent 
work demonstrating that current approaches to teaching the 
process of science are often ineffective (Abd-El-Khalick, 
2000; Moss, et al., 2001; Abd-El-Khalick 2006; Nadelson and 
Viskupic, 2010), often because introductory courses empha¬ 
size content rather than the process of science (Abd-El- 
Khalick, 2000; Bickmore et al., 2009). 

The shift to a skills-based curriculum offers new 
opportunities to adapt course offerings to the increasing 
variety of student needs. However, as our results demon¬ 
strate, doing so is not a panacea. Some of the most- 
challenging problems with more traditional curricula, such 
as how to help students develop a more advanced 
understanding of the nature of science, are not necessarily 
solved by focusing curricula on the integration of skills with 
content. But by highlighting the development of different 
skills across the entire curriculum, skills-based curricula 
make it easier to identify how different courses work (or do 
not work) together to create an integrated whole that 
ensures that students become skilled learners that can 
quickly adapt to rapidly evolving and expanding disciplines, 
both within geosciences and beyond. 
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APPENDIX A 

The baccalaureate origins of students earning doctorates in 
geosciences were determined as reported by the National 
Science Foundation Survey of Earned Doctorates program 
(http://webcaspar.nsf.gov/). For each institution included in 
the report, the total number of undergraduates going on to 
earn geoscience PhD degrees during the years 1966-2011 
was summed and expressed as the average number of 
earned doctorates per year (Table AI). To determine the 
institutional yield ratio, the average number of earned 
doctorates per year was divided by the institution's total 
undergraduate enrollment in 2010, as reported by the 
Integrated Postsecondary Education Data System of the 
National Center for Education Statistics (http://nces.ed.gov/ 
ipeds/deltacostproject/) (Table All). 
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TABLE AI. Total number of undergraduates going on to earn geoscience PhD degrees during the years 1966-2011. 


Rank 

Institution 

Doctorates/Year 

1 

University of California (Berkeley) 

6.67 

2 

University of Wisconsin (Madison) 

4.42 

3 

Stanford University (Stanford, CA) 

4.18 

4 

Carleton College (Northfield, MN) 

4.11 

5 

Pennsylvania State University (State College) 

4.00 

6 

Massachusetts Institute of Technology (Cambridge) 

3.71 

7 

University of California (Los Angeles) 

3.64 

8 

University of Washington (Seattle) 

3.64 

9 

Dartmouth College (Hanover, NH) 

3.58 

10 

Brown University (Providence, RI) 

3.47 

11 

University of California (Santa Barbara) 

3.33 

12 

University of Illinois (Urbana-Champaign) 

3.31 

13 

Cornell University (Ithaca, NY) 

3.29 

14 

University of Texas (Austin) 

3.29 

15 

Harvard University (Cambridge, MA) 

3.24 

16 

University of Colorado (Boulder) 

3.24 

17 

California Institute of Technology (Pasadena) 

3.22 

18 

University of Michigan (Ann Arbor) 

3.13 

19 

Colorado School of Mines (Golden) 

2.98 

20 

Franklin and Marshall College (Lancaster, PA) 

2.84 

21 

Princeton University (Princeton, NJ) 

2.84 

22 

University of Arizona (Tucson) 

2.80 

23 

University of Minnesota (Twin Cities) 

2.80 

24 

Indiana University (Bloomington) 

2.53 

25 

University of Rochester (Rochester, NY) 

2.51 






























678 C. E. Renshaw 


J. Geosci. Educ. 62, 668-678 (2014) 


TABLE AIL Institutional PhD yield ratio. Total number of undergraduates going on to earn geoscience PhD degrees during the 
years 1966-2011 normalized by undergraduate enrollment. 


Rank 

Institution 

Doctorates/Year/Thousand Undergraduates 

1 

Carleton College (Northfield, MN) 

2.05 

2 

California Institute of Technology (Pasadena) 

1.51 

3 

Franklin and Marshall College (Lancaster, PA) 

1.31 

4 

Amherst College (Amherst, MA) 

1.12 

5 

Pomona College (Claremont, CA) 

0.97 

6 

Beloit College (Beloit, WI) 

0.90 

7 

Colorado College (Colorado Springs) 

0.85 

8 

New Mexico Institute of Mining and Technology (Socorro, NM) 

0.80 

9 

Colgate University (Hamilton, NY) 

0.74 

10 

University of Montana (Missoula) 

0.74 

11 

Williams College (Williamstown, MA) 

0.73 

12 

Harvey Mudd College (Claremont, CA) 

0.70 

13 

Earlham College (Richmond, IN) 

0.61 

14 

Oberlin College (Oberlin, OH) 

0.61 

15 

Dartmouth College (Hanover, NH) 

0.60 

16 

Colorado School of Mines (Golden) 

0.58 

17 

Hamilton College (Clinton, NY) 

0.57 

18 

St. Lawrence University (Canton, NY) 

0.53 

19 

Haverford College (Haverford, PA) 

0.50 

20 

Occidental College (Los Angeles, CA) 

0.49 

21 

Juniata College (Huntingdon, PA) 

0.48 

22 

Whitman College (Walla Walla, WA) 

0.48 

23 

Allegheny College (Meadville, PA) 

0.48 

24 

Macalester College (St. Paul, MN) 

0.47 

25 

Cornell College (Mt. Vernon, IA) 

0.41 






























